Passive touch system and method of detecting user input

ABSTRACT

A method of tracking an object of interest preferably includes (i) acquiring a first image and a second image representing different viewpoints of the object of interest; (ii) processing the first image into a first image data set and the second image into a second image data set; (iii) processing the first image data set and the second image data set to generate a background data set associated with a background; (iv) generating a first difference map by determining differences between the first image data set and the background data set and a second difference map by determining differences between the second image data set and the background data set; (v) detecting a first relative position of the object of interest in the first difference map and a second relative position of the object of interest in the second difference map; and (vi) producing an absolute position of the object of interest from the first and second relative positions of the object of interest.

This application is a continuation of U.S. patent application Ser. No.12/118,407, filed May 9, 2008, which is a continuation of U.S. patentapplication No. 11/248,166, filed Oct. 13, 2005 (abandoned), which is acontinuation of U.S. patent application Ser. No. 10/408,671, filed Apr.8, 2003 (abandoned), which is a continuation of U.S. patent applicationSer. No. 09/610,481, filed Jul. 5,2000, now U.S. Pat. No. 6,803,906,which issued Oct. 12, 2004, all of which are incorporated herein byreference.

FIELD OF THE INVENTION

The present invention related generally to input devices and inparticular to a passive touch system and method of detecting use input.

BACKGROUND OF THE INVENTION

Touch systems are well known in the art and typically include a touchscreen having a touch surface on which contacts are made using a pointerin order to generate user input. Pointer contacts with the touch surfaceare detected and are used to generate corresponding output depending onareas of the contact surface where the contacts are made. There arebasically two general types of touch systems available and they can bebroadly classified as “active” touch systems and “passive” touchsystems.

Active touch systems allow a user to generate user input by contactingthe touch surface with a special pointer that usually requires some formof on-board power source, typically batteries. The special pointer emitssignals such as infrared light, visible light, ultrasonic frequencies,electromagnetic frequencies, etc. that activate the touch surface.

Passive touch systems allow a user to generate user input by contactingthe touch surface with a passive pointer and do not require the use of aspecial pointer in order to activate the touch surface. A passivepointer can be a finger, a cylinder of some material, or any suitableobject that can be used to contact some predetermined area of intereston the touch surface.

Passive touch systems provide advantages over active touch systems inthat any suitable pointing device, including a user's finger, can beused as a pointer to contact the touch surface. As a result, user inputcan easily be generated. Also, since special active pointers are notnecessary in passive touch systems, battery power levels and/or pointerdamage, theft, or misplacement are of no concern to users.

Passive touch systems have a number of applications relating to computeroperation and video display. For example, in one interactiveapplication, as is disclosed in U.S. Pat. No. 5,448,263 to Martin,assigned to the assignee of the present invention, the passive touchsystem is coupled to a computer and the computer display is projectedonto the touch surface of the touch screen. The coordinates representingspecific locations on the touch surface are mapped to the computerdisplay. When a user contacts the touch surface, the coordinates of thecontact are fed back to the computer and mapped to the computer displaythereby allowing the user to operate the computer in a manner similar tousing a computer mouse simply by contacting the touch surface.Furthermore, the coordinates fed back to the computer can be recorded inan application and redisplayed at a later time. Recording contactcoordinates is typically done when it is desired to record informationwritten or drawn on the touch surface by the user.

The resolution of a passive touch screen determines if the touch systemis suitable for recording information written or drawn on the touchscreen or only useful for selecting areas on the touch screen mapped tolarge regions on the computer or video display in order to manipulatethe computer or video display. Resolution is typically measured in dotsper inch (DPI). The DPI is related to the size of the touch screen andthe sampling ability of the touch system hardware and software used todetect contacts on the touch surface.

Low-resolution passive touch screens only have enough DPI to detectcontacts on the touch surface within a large group of pixels displayedby the computer or video system. Therefore, these low-resolution passivetouch screens are useful only for manipulating the computer or videodisplay.

On the other hand, high-resolution passive touch screens have sufficientDPI to detect contacts that are proportional to a small number of pixelsor sub-pixels of the computer or video display. However, a requirementfor high-resolution touch screens is the ability to detect when thepointer is in contact with the touch surface. This is necessary forwriting, drawing, mouse-click operations, etc. Without the ability todetect pointer contact with the touch screen, writing and drawing wouldbe one continues operation, and mouse clicks would not be possiblethereby making computer display manipulation impossible. A secondaryrequirement is the ability to detect when the pointer is “hovering”above the touch surface. Although not required for writing or drawing,today's computer operating systems are increasingly using hoverinformation to manipulate computer or video displays or pop-upinformation boxes.

Passive touch screens are typically either of the analog resistive type,Surface Acoustic Wave (SAW) type or capacitive type. Unfortunately,these touch screens suffer from a number of problems or shortcomings aswill be described.

Analog resistive touch screens typically have a high-resolution.

Depending on the complexity of the touch system, the resolution of thetouch screen can produce 4096×4096 DPI or higher. Analog resistive touchscreens are constructed using two flexible sheets that are coated with aresistive material and arranged as a sandwich. The sheets do not comeinto contact with each other until a contact has been made. The sheetsare typically kept separated by insulating microdots or by an insulatingair space. The sheets are constructed from ITO, which is mostlytransparent. Thus, the touch screen introduces some image distortion butvery little parallax.

During operation of an analog resistive passive touch screen, a uniformvoltage gradient is applied in one direction along a first of thesheets. The second sheet measures the voltage along the first sheet whenthe two sheets contact one another as a result of a contact made on thetouch surface. Since the voltage gradient of the first sheet can betranslated to the distance along the first sheet, the measured voltageis proportional to the position of the contact on the touch surface.When a contact coordinate on the first sheet is acquired, the uniformvoltage gradient is then applied to the second sheet and the first sheetmeasures the voltage along the second sheet. The voltage gradient of thesecond sheet is proportional to the distance along the second sheet.These two contact coordinates represent the X-Y position of the contacton the touch surface in a Cartesian coordinate system.

Since mechanical pressure is required to bring both sheets into contact,analog resistive touch screens can only detect contact when there issufficient pressure to bring the two sheets together. Analog resistivepassive touch screens cannot sense when a pointer is hovering over thetouch surface. Therefore, contact events and positions can only bedetected when actual contacts are made with the touch surface.

Surface Acoustic Wave (SAW) touch screens typically provide for mediumresolution and are not suitable for recording good quality writing. SAWtouch screens employ transducers on the borders of a glass surface tovibrate the glass and produce acoustic waves that ripple over the glasssurface. When a contact is made on the glass surface, the waves reflectback and the contact position is determined from the signature of thereflected waves.

Unfortunately, SAW touch screens exhibit noticeable parallax due to thethickness of the vibrating glass which is placed over the surface of thevideo or computer display. Also, contact events and positions can onlybe detected when actual contacts are made with the glass surface.Furthermore, SAW touch screens do not scale beyond a few feet diagonal.

Capacitive touch screens provide for low resolution because contacts canonly be determined in large areas (approximately ½″×½″). As a result,capacitive touch screens cannot be used for recording writing or drawingand are suitable for selecting areas on the touch screen correspondingto computer generated buttons displayed on the video or computerdisplay. These touch screens also suffer disadvantages in that they aresensitive to temperature and humidity. Similar to analog resistive touchscreens and SAW touch screens, capacitive touch screens can also onlydetect contact events and positions when actual contacts are made withthe touch surface.

Scalability of passive touch screens is important since the demand forlarger electronic digitizers is increasing. Where digitizers were oncesmall desktop appliances, today they have found there way ontoelectronic whiteboarding applications. The need to build a passive touchsensitive “wall” has become a requirement for new touch screenapplications. Existing passive touch screens of the types discussedabove are all limited in the maximum size where they are stillfunctional.

As will be appreciated, improvements to passive touch systems aredesired. It is therefore an object of the present invention to provide anovel passive touch system and method of detecting user input.

SUMMARY OF THE INVENTION

According to one aspect of the present invention there is provided apassive touch system comprising:

a passive touch surface;

at least two cameras associated with said touch surface, said at leasttwo cameras acquiring images of said touch surface from differentlocations and having overlapping fields of view; and

a processor receiving and processing images acquired by said at leasttwo cameras to detect the existence of a pointer therein and todetermine the location of said pointer relative to said touch surface.

In a preferred embodiment, the at least two cameras are two-dimensionalimage sensor and lens assemblies having fields of view looking along theplane of the touch surface. The processor determines the location of thepointer relative to the touch screen using triangulation. The processoralso determines when the pointer is in contact with the touch surfaceand when the pointer is hovering above the touch surface.

In one embodiment, the processor selects pixel subsets of imagesacquired by the image sensor and lens assemblies and processes the pixelsubsets to determine the existence of the pointer. The processorincludes a digital signal processor associated with each image sensorand lens assembly and a master digital signal processor in communicationwith the digital signal processors. The digital signal processorsassociated with each image sensor and lens assembly select the pixelsubsets and process the pixel subsets to determine the existence of thepointer. The master digital signal processor receives pixelcharacteristic data from the digital signal processors and triangulatesthe pixel characteristic data to determine the location of the pointerrelative to the touch surface.

According to another aspect of the present invention there is provided amethod of detecting the position of a pointer relative to a passivetouch surface comprising the steps of:

acquiring images of said touch surface from different locations usingcameras having overlapping fields of view; and

processing said images to detect the existence of a pointer therein andto determine the location of said pointer relative to said touchsurface.

The present invention provides advantages in that the passive touchsystem is of high resolution and allows actual pointer contacts with thetouch surface as well as pointer hovers above the touch surface to bedetected and corresponding output generated. Also, the present passivetouch system provides advantages in that it does not suffer fromparallax, image distortion, pointer position restrictions, imageprojection and scalability problems that are associated with prior artpassive touch systems.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the present invention will now be described more fullywith reference to the accompanying drawings in which:

FIG. 1 is a schematic diagram of a passive touch system in accordancewith the present invention;

FIG. 2 is an isometric view of a touch screen forming part of thepassive touch system of FIG. 1;

FIG. 3 is an isometric view of a corner portion of the touch screen ofFIG. 2;

FIG. 4 is a schematic diagram of a camera assembly forming part of thetouch screen of FIG. 2;

FIG. 5 is a front lavational view of the touch screen of FIG. 2 showingthe fields of view of two camera assemblies;

FIG. 6 is a schematic diagram of a master controller forming part of thepassive touch system of FIG. 1;

FIG. 7 is case diagram showing functions executed by the cameraassemblies;

FIG. 8 is a flowchart showing the steps performed during execution of afindPointerMotion() function;

FIG. 9 is a flowchart showing the steps performed during execution of anautoSelectThres() function;

FIG. 10 is a flowchart showing the steps performed during execution ofan extractPointer function;

FIG. 11 is a flowchart showing the steps performed during execution of acenterOfMass() function;

FIG. 12 is a flowchart showing the steps performed during execution of aprocessROI() function;

FIG. 13 is a flowchart showing the steps performed during execution of agetHighestRegion() function;

FIG. 14 shows an acquired image and a pixel subset of the image that isprocessed;

FIG. 15 shows a region of interest within the pixel subset;

FIG. 16 shows the triangulation geometry used to calculate a pointercontact position on the touch surface of the touch screen;

FIG. 17 shows an image acquired by an image sensor and lens assemblyincluding the pointer and its median line; and

FIG. 18 shows pointer contact and pointer hover for differentorientations of the pointer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Turning now to FIG. 1, a passive touch system in accordance with thepresent invention is shown and is generally indicated to by referencenumeral 50. As can be seen, passive touch system 50 includes a touchscreen 52 coupled to a master controller 54. Master controller 54 isalso coupled to a computer 56. Computer 56 executes one or moreapplication programs and provides display output that is projected ontothe touch screen 52 via a projector 58. The touch screen 52, mastercontroller 54, computer 56 and projector 58 form a closed-loop so thatuser contacts with the touch screen 52 can be recorded as writing ordrawing or used to control execution of application programs executed bythe computer 56.

FIGS. 2 to 4 better illustrate the touch screen 52. Touch screen 52includes a touch surface 60 bordered by a frame 62. Touch surface 60 ispassive and is in the form of a rectangular planar sheet of material.Camera subsystems are associated with each corner of the touch screen52. Each camera subsystem includes a camera assembly 63 mounted adjacenta different corner of the touch screen 52 by a frame assembly 64. Eachframe assembly 64 includes an angled support plate 66 on which thecamera assembly 63 is mounted. Supporting frame elements 70 and 72 aremounted on the plate 66 by posts 74 and secure the plate 66 to the frame62.

Each camera assembly 63, in this embodiment, includes a camera in theform of a two-dimensional CMOS camera image sensor and associated lensassembly 80, a first-in-first-out (FIFO) buffer 82 coupled to the imagesensor and lens assembly 80 by a data bus and a digital signal processor(DSP) 84 coupled to the FIFO 82 by a data bus and to the image sensorand lens assembly 80 by a control bus. A boot EPROM 86 and a powersupply subsystem 88 are also included.

In the present embodiment, the CMOS camera image sensor is a PhotobitPB300 image sensor configured for a 20×640 pixel subarray that can beoperated to capture image frames at rates in excess of 200 frames persecond. The FIFO buffer 82 is manufactured by Cypress under part numberCY7C4211V and the DSP 84 is manufactured by Analog Devices under partnumber ADSP2185M.

The DSP 84 provides control information to the image sensor and lensassembly 80 via the control bus. The control information allows the DSP84 to control parameters of the image sensor and lens assembly 80 suchas exposure, gain, array configuration, reset and initialization. TheDSP 84 also provides clock signals to the image sensor and lens assembly80 to control the frame rate of the image sensor and lens assembly 80.

As shown in FIG. 5, each image sensor and lens assembly 80 has a 55°field of view. The angle of the plate 66 is selected so that the fieldof view of each image and lens assembly 80 includes at least themajority of a different peripheral edge of the touch surface 60. In thisway, the entire touch surface 60 is within the fields of view of theimage sensor and lens assemblies 80.

Master controller 54 is best illustrated in FIG. 6 and includes a DSP90, a boot EPROM 92, a serial line driver 94 and a power supplysubsystem 95. The DSP 90 communicates with the DSPs 84 of the cameraassemblies 63 over a data bus through a serial port 96 and communicateswith the computer 56 over a data bus through a serial port 98 and theserial line driver 94. In this embodiment, the DSP 90 is alsomanufactured by Analog Devices under part number ADSP2185M. The serialline driver 94 is manufactured by Analog Devices under part numberADM222.

The master controller 54 and each camera assembly 63 follow acommunication protocol that enables bi-directional communications via acommon serial cable similar to a universal serial bus (USB). Thetransmission bandwidth is divided into thirty-two (32) 16-bit channels.Of the thirty-two channels, five (5) channels are assigned to each ofthe DSPs 84 in the camera assemblies 63 and to the DSP 90 in the mastercontroller 54 and the remaining seven (7) channels are unused. Themaster controller 54 monitors the twenty (20) channels assigned to thecamera assembly DSPs 84 while the DSPs 84 in the camera subsystems 63monitor the five (5) channels assigned to the master controller DSP 90.Communications between the master controller 54 and the cameraassemblies 63 are performed as background processes in response tointerrupts.

The general operation of the passive touch system 50 will now bedescribed. Each camera assembly 63 acquires images of the touch surface60 within the field of view of its image sensor and lens assembly 80 atthe frame rate established by the DSP clock signals and processes theimages to determine if a pointer is in the acquired images. If a pointeris in the acquired images, the images are further processed to determinecharacteristics of the pointer contacting or hovering above the touchsurface 60. Pointer characteristics are then converted into pointerinformation packets (PIPs) and the PIPs are queued for transmission tothe master controller 54.

The camera assemblies 63 also receive and respond to diagnostic PIpsgenerated by the master controller 54.

The master controller 54 polls the camera assemblies 63 at a setfrequency (in this embodiment 70 times per second) for PIPs andtriangulates pointer characteristics in the PIPs to determine pointerposition data. The master controller 54 in turn transmits pointerposition data and/or status information to the personal computer 56. Inthis manner, the pointer position data transmitted to the personalcomputer 56 can be recorded as writing or drawing or can be used tocontrol execution of application programs executed by the computer 56.The computer 56 also updates the display output conveyed to theprojector 58 so that information projected onto the touch surface 60reflects the pointer activity.

The master controller 54 also receives commands from the personalcomputer 56 and responds accordingly as well as generates and conveysdiagnostic PIPs to the camera assemblies 63.

Specifics concerning the processing of acquired images and thetriangulation of pointer characteristics in PIPs will now be describedwith particular reference to FIGS. 7 to 13.

Initially, an alignment routine is performed to align the image sensorand lens assemblies 80. During the alignment routine, a pointer is heldin the approximate center of the touch surface 60. Subsets of the pixelsof the image sensor and lens assemblies 80 are then selected until asubset of pixels for each image sensor and lens assembly 80 is foundthat captures the pointer and the pointer tip on the touch surface 60.This alignment routine allows for a relaxation in mechanical mounting ofthe image sensor and lens assemblies on the frame assemblies 64. Theidentification of the pointer tip on the touch surface 60 also gives acalibration that determines the row of pixels of each image sensor andlens assembly 80 that detects actual contacts made with the touchsurface. Knowing these pixel rows allows the difference between pointerhover and pointer contact to be determined.

In this embodiment, since a computer display is projected onto the touchsurface 60, during the alignment routine several known coordinatelocations are also displayed and the user is required to touch thesecoordinate locations in sequence using the pointer so that the subset ofpixels for each of image sensor and lens assembly 80 includes all of thecoordinate locations as well. Calibration data is then stored forreference so that pointer contacts on the touch surface 60 can be mappedto corresponding areas on the computer display.

As mentioned above, each camera assembly 63 acquires images of the touchsurface 60 within its field of view. The images are acquired by theimage and lens assembly 80 at intervals in response to the clock signalsreceived from the DSP 84. Each image acquired by the image and lensassembly 80 is sent to the FIFO buffer 82.

The DSP 84 in turn reads each image from the FIFO buffer 82 andprocesses the image to determine if a pointer is located in the imageand if so, to extract the pointer and related pointer statisticalinformation. To avoid processing significant numbers of pixelscontaining no useful information, only the subset of the pixels in theimage determined during the alignment routine are actually processed asis shown in FIG. 14.

In order to determine if a pointer is located in the image and extractpointer and related pointer statistical information, the DSP 84 executesa main findPointerMotion( ) function 120 that calls a number of otherfunctions, namely an autoSelectThres() function 122, an extractPointerfunction 124, a centerOfMass() function 126, and a processROI() function128 (see FIG. 7). The extractPointer() function 128 also calls agetHighestRegion() function 130.

The findPointerMotion() function 120 is used to extract the pointer fromthe image. Turning now to FIG. 8, the steps performed during executionof the findPointerMotion() function 120 is shown. When thefindPointerMotion() function is called, a check is made to determine ifa previous image iPrev including a pointer exists (step 150). If noprevious image iPrev exists, center of mass parameters Cx and Cz areassigned zero values (step 152). The current image iCurr being processedis then designated as the previous image iPrev (step 154) to completethe findPointerMotion() function.

At step 150, if a previous image iPrev exists, the current image iCurris subtracted from the previous image iPrev and the absolute value ofthe difference image iDiff is taken (step 156). By forming thedifference image iDiff, background features and noise are removed. TheautoSelectThres() function 122 is then called to select a thresholdvalue tValue for the difference image iDiff (step 158) based on thehistogram of the difference image iDiff. The threshold iThres of thedifference image iDiff is then taken (step 160) to highlight further thepointer within the current image iCurr. During thresholding a grayscaleimage is mapped to the binary difference image iDiff. Pixels in thedifference image with values equal to or less than the threshold valuetValue are made black while all other pixels are made white. The resultis a binary image containing the pointer and some noise both designatedby white pixels.

Once the difference image has been thresholded, the extractPointerfunction 124 is called (step 162) to extract the pointer ptr from thedifference image iDiff and ignore the noise. The size of the pointer ptris then examined to determine if it is greater than a threshold valueMIN_PTR_SIZE (step 164).

If the size of the pointer is greater than the threshold valueMIN_PTR_SIZE, the centerOfMass() function 126 is called (step 166) todetermine the center of the pointer. Following this, the processROI()function 128 is called (step 168) to select a region of interest ROIwithin the difference image iDiff and extract the pointer from theregion of interest.

Once the pointer has been extracted from the region of interest ROI, thesize of the extracted pointer is determined (step 170). If the pointersize is greater than zero, a medianLine function is called (step 172).During execution of the medianLine function, the median line of thepointer (i.e. the pointer location within the region of interest) iscalculated using linear least squares. The current image iCurr is thendesignated as the previous image iPrev to complete thefindPointerMotion() function.

At step 164, if the pointer size is equal to zero, the center of massparameters Cx and Cz are examined (step 174). If both of the center ofmass parameters Cx and Cz have values greater zero, the processROI()function 128 is called (step 168) to select a region of interest ROIwithin the difference image iDiff and extract the pointer from theregion of interest. At step 174, if one or both of the center of massparameters Cx and Cz have values equal to zero or at step 170, if thesize of the pointer is less than zero, a no pointer found condition isdetermined (step 176). At this stage, the current image iCurr isdesignated as a calibration image glRef. The findPointerMotion()function then proceeds to step 152 where the center of mass parametersCx and Cz are assigned zero values.

As mentioned above, at step 158, when the findPointerMotion() function120 calls the autoSelectThres() function 122, a threshold value for thedifference image iDiff is selected based on the histogram of thedifference image so that when the difference image iDiff is thresholded,the pointer is further highlighted from background features and noise.Selection of the threshold value in this manner is more robust thanhardcoding the threshold value.

Turning now to FIG. 9, the steps performed during execution of theautoSelectThres() function 122 are illustrated. As can be seen, in orderto select the threshold level, a histogram of the difference image iDiffis taken and the number of pixels in each bin of the histogram arecounted (step 200). The number of pixels in the bin having the highestcount is used as a peak parameter and the threshold value is initiallyassigned a value of one (step 202). The number of bins having non-zerocounts is then examined to determine if more than eight (8) bins havenon-zero counts (step 204). If less than eight (8) bins have non-zerocounts, the threshold value remains at its initially assigned value andthe autoSelectThres() function is completed.

At step 204, if more than eight (8) bins have non-zero counts, thenumber of non-zero bins is checked again to determine if an entiredifference image is being processed (i.e. the autoSelectThres() functionwas called by the findPointerMotion() function 120) or if a region ofinterest ROI within the difference image is being processed (i.e. theautoSelecThres() function was called by the processROI() function 128)(step 206). If the entire difference image iDiff is being processed, athreshold minimum parameter (tMin) is set to a value of twelve (12) anda Peak_Div parameter is set to a value of eight (8) (step 208). Aminimum count parameter minCount is then calculated by dividing the peakparameter determined at step 202 by the Peak_Div parameter (step 210).If a region of interest is being processed, the threshold minimumparameter (tMin) is set to a value of forty (40) and the Peak_Divparameter is set to a value of thirty-two (32) (step 212) beforeproceeding to step 210.

Once minCount has been determined, the peak level is checked todetermine if it is greater than the threshold minimum tmin (step 214).Peak level is the grayscale level that contains the most pixels. In thecase of a tie, the grayscale level with the highest numerical value(i.e. the closest to 255) is chosen. If the peak level is greater thanthe threshold minimum tMin, a startLevel parameter is assigned a valueequal to the peak level+1 (step 216). At step 214, if the peak level isless than the threshold minimum tMin, the startLevel parameter isassigned a value equal to the threshold minimum tMin (step 218).

Following step 216 or 218, a loop is entered. During the loop, thelevCount for each bin having a bin number between the value of thestartLevel parameter and two hundred and fifty-five (255) is examined todetermine if it is greater than zero and if it is less than the minCountparameter determined at step 210 (step 220). If the condition is metduring the loop, the loop is exited and the threshold value is assigneda value equal to the bin number having the levCount that resulted in theloop being exited+1 (step 222). If the condition is not met, the loop isexited after the levCount for bin number 255 has been checked.

Once the loop has been exited, the threshold value is checked todetermine if it is less than the minimum threshold value tMin (step224). If not, a check is again made to determine if an entire differenceimage is being processed or whether a region of interest ROI is beingprocessed (step 226). If the threshold value is less than the minimumthreshold value tMin, the threshold value is set equal to the minimumthreshold value tMin (step 228) before the check is made to determine ifan entire difference image is being processed or whether a region ofinterest is being processed (step 226).

At step 226, if a difference image iDiff is being processed, theautoSelectThres() function is completed. However, if a region ofinterest is being processed, a parameter p is assigned a valuecorresponding to the first grayscale level at which 90% or more of thepixels will go black (step 230). The parameter p is then compared to thethreshold level (step 232). If the parameter p is less than thethreshold level, the autoSelectThres() function is completed. If theparameter p is greater than the threshold level, the threshold value isset to the value of parameter p (step 234) and the autoSelectThres()function is completed.

As mentioned above, at step 162 the findPointerMotion() function 120calls the extractPointer() function 124 to extract the pointer from thebinary image and ignore the noise. This is done by selecting the “whiteregion” in the binary image that is greater than or equal to a certainminimum size and is the highest region (i.e. the largest in the y-axis(20 pixel axis)). Specifically, when the extractPointer() function 124is called, the extractPointer() function calls the getHighestRegion()function 130 (step 250). The getHighestRegion() function 130 uses thethreshold value and tol parameters to select the appropriate whiteregion szRegion in the thresholded difference image. The tol parameteris used to avoid situations where board surface noise is mistaken as apointer. FIG. 13 shows the steps performed during this function.

Once the white region szRegion has been selected, the white regionszRegion is checked to see if it is greater than zero (step 252). Ifnot, a no pointer condition is determined (step 254). If the whiteregion szRegion is greater than zero, morphological operator of erosionand dilation are used on the white region to reduce further noise (steps256 to 264) and the extractPointer() function is completed.

As mentioned above, at step 166 the findPointerMotion() function 120calls the centerOfMass() function 126 to determine the center of thepointer. During this function, the black pixels in the binary image aretreated as having a mass of zero (0) and the white pixel are treated ashaving a mass of one (1). The physics formulae for center-of-mass areused. The equation below gives the center of mass in the x-direction:

C _(x)=sum((X _(i))/M)

where:

X_(i) are the x-coordinates of the white pixels in the binary image; and

M is the number of white pixels in the binary image.

Initially, once the centerOfMass() function is executed, the center ofmass parameters massX, massZ and a mass parameter are assigned zerovalues (see step 300 in FIG. 11). A loop is then entered to calculatethe center of mass parameters massX and massZ using the above equationand to calculate the mass parameter (step 302).

Upon exiting the loop, the mass parameter is checked to determine if itsvalue is greater than zero (step 304). If the value of the massparameter is equal to zero, the center of mass parameters Cx and Cz areassigned values of zero (step 306) and the centerOfMass() function 126is completed. At step 304, if the value of the mass parameter is greaterthan zero, the center of mass coordinates Cx and Cz are calculated (step308) using the equations:

Cx=massX/mass; and

Cz=massZ/mass.

Once the center of mass coordinates have been calculated, thecenterOfMass() function 126 is completed.

As mentioned above, at step 168 the findPointerMotion() function 120calls the processROI() function 128 to process the region-of-interest ina manner similar to the findPointerMotion() function 120 except, herethe image size is 100×20 pixels and a calibration image including onlybackground (i.e. no pointer) is used in place of the previous image.Upon execution of the processROI() function 128, xLeft and xRightparameters are calculated by subtracting and adding fifty (50) to thecenter of mass parameter Cx (step 350). The value of parameter xLeft isthen checked to determine if it is less than one (1) (step 352). If theparameter xLeft has a value less than one (1), the parameter xRight isrecalculated and the parameter xLeft is assigned a value of one (1)(step 354) to define boundaries of the region of interest as shown inFIG. 15. A difference image iDiff of the region of interest is thencalculated by subtracting the region of interest of the current imagefrom the region of interest of the calibration image glRef determined atstep 176 of the findPointerMotion() function 120 and taking the absolutevalue of the difference (step 356).

At step 352, if the parameter xLeft has a value greater than one (1),the parameter xRight is checked to determine if it has a value greaterthan 640 (step 358). If the parameter xRight has a value greater than640, the parameter xLeft is recalculated and the parameter xRight isassigned a value of one (1) (step 360) to define boundaries of theregion of interest. The processROI() function 128 then proceeds to step356 to calculate the difference image iDiff of the region of interest.At step 358, if the parameter xRight has a value less than 640, theprocessROI() function 128 proceeds directly to step 356 to calculate thedifference image iDiff of the region of interest.

Once the difference image iDiff of the region of interest has beencalculated, the autoSelectThres() function 122 is called to select athreshold value for the difference image iDiff of the region of interest(step 362) in the manner described above with reference to FIG. 9. Thedifference image iDiff of the region of interest is then thresholded(step 364). Following this, the extractPointer() function 124 is calledto extract the pointer from the difference image iDiff of the region ofinterest (step 366) in the manner described above with reference to FIG.10.

Once the acquired image has been processed in the above manner, a PIPfor the acquired image is created by the DSP 84. The PIP is a five (5)word packet and has a layout including camera identification, an LRCchecksum to ensure data integrity and a valid tag to ensure zero packetsare not valid. The valid tag indicates whether the PIP relates to apointer characteristic packet (10), a diagnostic packet for a specificcamera assembly 63 (01) or a diagnostic packet for all camera assemblies63 (11). Table 1 below shows the PIP layout.

TABLE 1 Word 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 PointerCharacteristics packet (generated by Camera) 0 Valid Camera X Intercept(at Y0) tag # 1 Frame rate intensity/color 2 Packet # pointer area 3Unused X intercept (at Y19) 4 Unused Z position LRC checksum DiagnosticPacket (generated by Camera or Master) 0 Valid Camera tag # 1 2 Packet #3 4 LRC checksum

As mentioned above, each camera assembly 63 acquires and processes animage in the manner described above in response to each clock signalgenerated by its DSP 84. The PIPs created by the DSPs 84 are only sentto the master controller 54 when the camera assemblies 63 are polled bythe master controller. The DSPs 84 create PIPs faster than the mastercontroller 54 polls the camera assemblies 63. PIPs that are not sent tothe master controller 54 are overwritten.

When the master controller 54 polls the camera assemblies 63, frame syncpulses are sent to the camera assemblies 63 to initiate transmission ofthe PIps created by the DSPs 84. Upon receipt of a frame sync pulse,each DSP 84 transmits the PIP to the master controller 54 over the databus. The PIPs transmitted to the master controller 54 are received viathe serial port 96 and auto-buffered into the DSP 90.

After the DSP 90 has polled the camera assemblies 63 and has receivedPIPs from each of the camera assemblies, the DSP 90 processes the PIPsusing triangulation to determine the location of the pointer relative tothe touch surface 60 in (x,y) coordinates. Specifically, the PIPs frompairs of camera assemblies 63 are processed using triangulation.

FIG. 16 shows that two angles φ₁ and φ₂ are needed to triangulate theposition (x₀,y₀) of the pointer relative to the touch screen 60. ThePIPs generated by each camera assembly 63 include a numberθε[0,sensorResolution−1] (see FIG. 17) identifying the median line ofthe pointer. The sensor Resolution, in the case of the Photobit PB300image sensor, is 640. The equations below relate the angle φ to theposition θ taking into account the field-of-view of the image sensor andlens assembly 80:

$\begin{matrix}{\varphi = {{\frac{\theta}{sensorResolution} \times F_{ov}} - \delta}} & (1) \\{\varphi = {{\frac{{SensorResolution} - \theta}{sensorResolution} \times F_{{ov}\;}} - \delta}} & (2)\end{matrix}$

The above equations subtract away an angle δ that allows the imagesensor and lens assembly 80 to have some overlap with the frame 62. Theoverlap with the frame 62 is desired due to mechanical tolerance issuesin the frame assemblies 64 (i.e. the angle of the plate 66 can have anerror of 1° to 2°). The angle δ is allowed to be negative, meaning thatthere is no overlap with the frame 62, in fact part of the touch surface60 along the frame 62 is missed. Equation 1 or 2 is used to determine .φ, depending on the mounting and/or optical properties of the imagesensor and lens assembly 80. If the image acquired by the cameraassembly 63 is rotated as a result of the mounting and/or opticalproperties of the image sensor and lens assembly 80, then equation 2 isused. Equation 1 is used otherwise. In the present embodiment, equation1 is used with the camera assemblies 63 positioned at the top left andbottom right comers of the touch screen 52 and equation 2 is used withthe camera assemblies 63 positioned at the bottom left and top rightcorners of the touch screen 52.

As discussed above, equations 1 and 2 allow the pointer median line dataincluded in the PIPs to be converted by the DSP 90 into an angle φ withrespect to the x-axis. When two such angles are available, theintersection of the median lines extending at these angles from theirrespective camera assemblies 63 yields the location of the pointerrelative to the touch surface 60.

In this embodiment, since the touch screen 52 includes four cameraassemblies 63, six pairs of camera assemblies can be used fortriangulation. The following discussion describes how a pointer positionis determined by triangulation for each pair of the camera assemblies63.

In order to determine a pointer position using the PIPs received fromthe camera assemblies 63 along the left side of the touch screen 52, thefollowing equations are used to determine the (x₀,y₀) coordinates of thepointer position given the angles φ₀ and φ₁ for the upper and lowercamera assemblies:

$\begin{matrix}{x_{0} = {\frac{h}{w} \times \frac{1}{{\tan \left( \varphi_{0} \right)} + {\tan \left( \varphi_{1} \right)}}}} & (3) \\{y_{0} = \frac{\tan \left( \varphi_{0} \right)}{{\tan \left( \varphi_{0} \right)} + {\tan \left( \varphi_{1} \right)}}} & (4)\end{matrix}$

where:

h is the height of the touch screen 52 i.e. the vertical distance fromcamera assembly focal point-to-focal point;

w is the width of the touch screen 52 i.e. the horizontal distance fromcamera assembly focal point-to-focal point; and

φ_(i) is the angle with respect to the horizontal, measured using cameraassembly i and equation 1 or 2.

For the camera assemblies 63 along on the right side of the touch screen52, the following equations are used to determine the (x₀, y₀)coordinates of the pointer position given the angles φ₂ and φ₃ for theupper and lower camera assemblies:

$\begin{matrix}{x_{0} = {1 - {\frac{h}{w} \times \frac{1}{{\tan \left( \varphi_{2} \right)} + {\tan \left( \varphi_{3} \right)}}}}} & (5) \\{y_{0} = {1 - \frac{\tan \left( \varphi_{2} \right)}{{\tan \left( \varphi_{2} \right)} + {\tan \left( \varphi_{3} \right)}}}} & (6)\end{matrix}$

The similarity between equations 3 and 5, i.e. equation 5=1-equation 3once φ₂ and φ₃ have been substituted into equation 3 for φ₁ and φ₂respectively should be apparent. Equations 4 and 6 are related in asimilar manner.

In order to determine a pointer position using the camera assemblies 63along the bottom of the touch screen 52, the following equations areused to determine the (x₀, y₀) coordinates of the pointer position giventhe angles φ₀ and φ₃ for bottom left and bottom right camera assemblies:

$\begin{matrix}{x_{0} = \frac{\tan \left( \varphi_{3} \right)}{{\tan \left( \varphi_{0} \right)} + {\tan \left( \varphi_{3} \right)}}} & (7) \\\begin{matrix}{y_{0} = {\frac{w}{h} \times \frac{\tan \left( \varphi_{3} \right)}{{\tan \left( \varphi_{0} \right)} + {\tan \left( \varphi_{3} \right)}} \times {\tan \left( \varphi_{0} \right)}}} \\{= {\frac{w}{h} \times x_{0} \times {\tan \left( \varphi_{0} \right)}}}\end{matrix} & (8)\end{matrix}$

In order to determine a pointer position using the camera assemblies 63along the top of the touch screen 52, the following equations are usedto determine the (x₀, y₀) coordinates of the pointer position given theangles φ₁ and φ₂ for the top left and top right camera assemblies:

$\begin{matrix}{x_{0} = \frac{\tan \left( \varphi_{2} \right)}{{\tan \left( \varphi_{1} \right)} + {\tan \left( \varphi_{2} \right)}}} & (9) \\\begin{matrix}{y_{0} = {1 - {\frac{w}{h} \times \frac{\tan \left( \varphi_{2} \right)}{{\tan \left( \varphi_{1} \right)} + {\tan \left( \varphi_{2} \right)}} \times {\tan \left( \varphi_{1} \right)}}}} \\{= {1 - {\frac{w}{h} \times x_{0} \times {\tan \left( \varphi_{1} \right)}}}}\end{matrix} & (10)\end{matrix}$

The similarity between equations 7 and 9, i.e. equation 9=equation 7once φ₁ and φ₂ have been substituted into equation 7 for φ₀ and φ₃should be apparent. Equations 8and 10 have the following relationship:equation 10=1-equation 8 once φ₁ and φ₂ have been substituted intoequation 8 for φ₀ and φ₃ respectively.

In order to determine a pointer position using the camera assemblies 63across the bottom left to top right corner diagonal, the followingequations are used to determine the (x₀,y₀) coordinates of the pointerposition given the angles φ₀ and φ₂ for bottom left and top right cameraassemblies:

$\begin{matrix}{x_{0} = \frac{\frac{h}{w} - {\tan \left( \varphi_{2} \right)}}{{\tan \left( \varphi_{0} \right)} - {\tan \left( \varphi_{2} \right)}}} & (11) \\{y_{0} = {\frac{1 - \frac{w}{h} - {\tan \left( \varphi_{2} \right)}}{{\tan \left( \varphi_{0} \right)} - {\tan \left( \varphi_{2} \right)}} \times {\tan \left( \varphi_{0} \right)}}} & (12)\end{matrix}$

In order to determine a pointer position using the camera assemblies 63across the bottom right to top left diagonal, the following equationsare used to determine the (x₀,y₀) coordinates of the pointer positiongiven the angles φ₁ and φ₃ for the bottom right and top left cameraassemblies:

$\begin{matrix}{x_{0} = \frac{\frac{h}{w} - {\tan \left( \varphi_{3} \right)}}{{\tan \left( \varphi_{1} \right)} - {\tan \left( \varphi_{3} \right)}}} & (13) \\{y_{0} = {1 - {\frac{1 - \frac{w}{h} - {\tan \left( \varphi_{3} \right)}}{{\tan \left( \varphi_{1} \right)} - {\tan \left( \varphi_{3} \right)}} \times {\tan \left( \varphi_{1} \right)}}}} & (14)\end{matrix}$

The similarity between equations 11 and 13, i.e. equation 13=equation 11once °1 and ° 3 have been substituted into equation 11 for φ₀ and φ₂should be apparent. Equations 12 and 14 have the following relationship:equation 14=1-equation 12 once φ₁ and φ₃ have been substituted intoequation 12 for φ₀ and φ₂ respectively.

As will be appreciated, the above equations generate the coordinates x₀and y₀ on a scale of [0, 1]. Therefore, any appropriate coordinate scalecan be reported by multiplying x₀ and y₀ by the maximum X and maximum Yvalues respectively.

In the present embodiment, the DSP 90 calculates the pointer positionusing triangulation for each camera pair excluding the diagonal pairs.The resulting pointer positions are then averaged and the resultingpointer position coordinates are queued for transmission to the personalcomputer 56 via the serial port 98 and the serial line driver 94. Sincethe rows of pixels of the image sensor and lens assemblies 80 thatcorrespond to actual contacts with the touch surface 60 are known, anyZ-position in a PIP that does not correspond with one of these rows isby definition a pointer hover event.

If desired, pointer velocity and angle can be calculated by the DSP 90as shown in FIG. 18. The velocity of the pointer is calculated byexamining the changes in the Z-position (or X-intercept) of the pointerin successive PIPs and knowing the camera frame rate. For example, ifthe camera frame rate is 200 frames per second and the Z-positionchanges by 1 pixel per frame, the pointer velocity is 200 pixels persecond.

The angle of the pointer can be determined due to the fact that the PIPincludes the X-intercept at pixel rows 0 and 19 of the median line.Since the X distance (the difference between X-intercepts) and the Ydistance (the number of pixel rows) are known, all of the informationnecessary to calculate the pointer angle is available.

The present invention provides advantages in that the passive touchsystem 50 does not suffer parallax and/or image distortion problems dueto the fact that a glass or other transparent overlay over a computer orvideo display is not required. In addition, the present passive touchsystem 50 allows both pointer contact and pointer hover over the touchsurface 60 to be detected by using two-dimensional image sensor and lensassemblies 80 in the plane of the touch surface 60. Pointer contact withthe touch surface 60 is defined only when the pointer is in very closeproximity of the touch surface. The present invention also providesadvantages in that the pointer position with respect to the touchsurface is not restricted since the image sensor and lens assemblies 80look along the plane of the touch surface 60.

With respect to resolution, the resolution of the passive touch systemis a function of the distances of the pointer with respect to the imagesensor and lens assemblies 80, the number of pixel elements in the imagesensor and lens assemblies and the fields of view of the image sensorand lens assemblies. Since image sensor and lens assemblies areavailable with pixel elements that range in number from tens of thousandto many millions and since the number of pixel elements in image sensorsand lens assemblies of this nature is only expected to increase, theresolution of the present passive touch system 50 is high.

The passive touch system 50 also provides advantages in that alignmentis automatically corrected since only pixel subsets of images thatinclude the touch surface and the pointer are processed. In addition,the present passive touch system allows for very fast acquisition ofimage data since the image sensor and lens assemblies can be triggeredto capture images at rates exceeding two hundred frames per second.

The present passive touch system 50 is scalable and can include a touchsurface 60 of arbitrary size. When used in conjunction with a projectedcomputer image, the number of pixels of the image sensor and lensassemblies should be proportional to the number of pixels beingdisplayed on the touch surface 60. For example, if a projected computerimage is 1024×768 pixels, the size of the projected image is not be ofconcern provided the image sensor and lens assemblies 80 are able toresolve the (x,y) coordinates with sufficient accuracy with respect tothe displayed pixels.

Although the passive touch system 50 is shown including cameraassemblies 63 associated with each corner of the touch screen 52, thoseof skill in the art will appreciate that only two camera assemblies arerequired. In this case, the fields of view of the image sensor and lensassemblies are preferably selected so that the entire touch surface 60is encompassed since the locations of pointer contacts are determinedonly when they occur within the overlapping fields of view of the cameraassemblies 63.

Also, although the passive touch system 50 is described as including aprojector to display the computer display output onto the touch surface60, this is not required. No information need be displayed on the touchsurface.

Although a preferred embodiment of the present invention has beendescribed, those of skill in the art will appreciate that variations andmodifications may be made without departing from the spirit and scopethereof as defined by the appended claims.

1. A method of tracking an object of interest controlled by a user tointerface with a computer, the method comprising: acquiring images fromat least two viewpoints; processing the acquired images to produce animage data set for each acquired image; comparing each image data set toone or more background data sets to produce a difference map for eachacquired image; detecting a relative position of an object of interestwithin each difference map; producing an absolute position of the objectof interest from the relative positions of the object of interest; andusing the absolute position to allow the user to interact with acomputer application.
 2. The method of claim 1 further including:mapping the absolute position of the object of interest to screencoordinates associated with the computer application; and using themapped position to interface with the computer application.
 3. Themethod of claim 1 further including: recognizing a gesture associatedwith the object of interest by analyzing changes in the absoluteposition of the object of interest; and combining the absolute positionand the gesture to interface with the computer application.
 4. Amultiple camera tracking system for interfacing with an applicationprogram running on a computer, the tracking system comprising: two ormore video cameras arranged to provide different viewpoints of a regionof interest and operable to produce a series of video images; aprocessor operable to receive the series of video images and detectobjects appearing in the region of interest, the processor executing aprocess to: generate a background data set from the video images;generate an image data set for each received video image; compare eachimage data set to the background data set to produce a difference mapfor each image data set; detect a relative position of an object ofinterest within each difference map; and produce an absolute position ofthe object of interest from the relative positions of the object ofinterest and map the absolute position to a position indicatorassociated with the application program.
 5. The multiple camera trackingsystem of claim 4 wherein the region of interest is defined to be infront of a video display associated with the computer, and wherein theprocessor is further operable to map the absolute position of the objectof interest to the position indicator such that the location of theposition indicator on the video display is aligned with the object ofinterest.
 6. The multiple camera tracking system of claim 4 wherein theprocessor is further configured to emulate a computer mouse function. 7.The multiple camera tracking system of claim 4 wherein the backgrounddata set includes data points representing at least a portion of astationary structure.
 8. A multiple camera tracking system forinterfacing with an application program running on a computer, thetracking system comprising: two or more video cameras arranged toprovide different viewpoints of a region of interest and operable toproduce a series of video images; a processor operable to receive theseries of video images and detect objects appearing in the region ofinterest, the processor executing a process to: generate a backgrounddata set from the video images; generate an image data set for eachreceived video image; compare each image data set to the background dataset to produce a difference map for each image data set; detect arelative position of an object of interest within each difference map;produce an absolute position of the object of interest from the relativepositions of the object of interest; define sub regions within theregion of interest; identify a sub region occupied by the object ofinterest; associate an action with the identified sub region that isactivated when the object of interest occupies the identified subregion; and apply the action to interface with the application program.